Method for detecting a clear path of travel for a vehicle enhanced by object detection

ABSTRACT

Method for detecting a clear path of travel for a host vehicle including fusion of clear path detection by image analysis and detection of an object within an operating environment of the host vehicle including monitoring an image from a camera device, analyzing the image through clear path detection analysis to determine a clear path of travel within the image, monitoring sensor data describing the object, analyzing the sensor data to determine an impact of the object to the clear path, utilizing the determined impact of the object to describe an enhanced clear path of travel, and utilizing the enhanced clear path of travel to navigate the host vehicle.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.12/581,659 filed on Oct. 19, 2009, which is a continuation-in-part ofU.S. application Ser. No. 12/474,594 filed on May 29, 2009, which is acontinuation-in-part of U.S. application Ser. No. 12/108,581 filed onApr. 24, 2008. U.S. application Ser. No. 12/581,659 filed on Oct. 19,2009 claims the benefit of U.S. Provisional Application 61/215,745 filedon May 8, 2009. U.S. application Ser. No. 12/581,659, U.S. applicationSer. No. 12/474,594, U.S. application Ser. No. 12/108,581, and U.S.Provisional Application 61/215,745 are incorporated herein by reference.

TECHNICAL FIELD

This disclosure is related to automated or semi-automated control of amotor vehicle.

BACKGROUND

The statements in this section merely provide background informationrelated to the present disclosure and may not constitute prior art.

Autonomous driving systems and semi-autonomous driving systems utilizeinputs regarding the road and other driving conditions to automaticallycontrol throttle and steering mechanisms. Accurate estimation andidentification of a clear path over which to operate a motor vehicle iscritical to successfully replacing the human mind as a control mechanismfor vehicle operation.

Road conditions can be complex. Under normal operation of a vehicle, thehuman operator makes hundreds of observations per minute and adjustsoperation of the vehicle on the basis of perceived road conditions. Oneaspect of perceiving road conditions is the perception of the road inthe context of objects in and around the roadway and navigating a clearpath through any objects. Replacing human perception with technologypreferentially includes some means to accurately perceive objects, forexample, including stationary objects such as roadside curbs and movingobjects such as other vehicles, and road conditions, such as lanemarkers, potholes, or icy patches upon the roadway, and continue toeffectively navigate around such navigational concerns.

Technological means for perceiving an object or road conditions includedata from visual cameras, radar imaging, LIDAR, ultrasonic sensors,vehicle to vehicle communications, vehicle to infrastructurecommunications, and use of global positioning data with a digital map.Cameras translate visual images in the form of radiation such as lightpatterns or infrared signatures into a readable data format. One suchdata format includes pixelated images, in which a perceived scene isbroken down into a series of pixels. Radar imaging utilizes radio wavesgenerated by a transmitter to estimate shapes and objects present infront of the transmitter. Patterns in the waves reflecting off theseshapes and objects can be analyzed and the locations of objects can beestimated.

Once data has been generated regarding the ground in front of thevehicle, the data must be analyzed to estimate the presence of objectsor road conditions from the data. By using cameras, radar imagingsystems, LIDAR, and other methods, ground or roadway in front of thevehicle can be analyzed for the presence of objects or road conditionsthat might need to be avoided. However, the mere identification ofpotential navigational concerns to be avoided does not complete theanalysis. An important component of any autonomous system includes howpotential navigational concerns identified in perceived ground data areprocessed and manipulated to identify a clear path in which to operatethe vehicle.

One known method to identify a clear path in which to operate thevehicle is to catalog and provisionally identify all perceivednavigational concerns and identify a clear path in light of thelocations and behaviors of identified concerns. Images may be processedto identify and classify navigational concerns according to their formand relationship to the roadway. While this method can be effective inidentifying a clear path, it requires a great deal of processing power,for example, requiring the recognition and separation of differentobjects in the visual image, for instance, distinguishing between a treealong the side of the road and a pedestrian walking toward the curb.Such methods can be slow or ineffective to process complex situations ormay require bulky and expensive equipment to supply the necessaryprocessing capacity.

SUMMARY

Method for detecting a clear path of travel for a host vehicle includingfusion of clear path detection by image analysis and detection of anobject within an operating environment of the host vehicle includingmonitoring an image from a camera device, analyzing the image throughclear path detection analysis to determine a clear path of travel withinthe image, monitoring sensor data describing the object, analyzing thesensor data to determine an impact of the object to the clear path,utilizing the determined impact of the object to describe an enhancedclear path of travel, and utilizing the enhanced clear path of travel tonavigate the host vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments will now be described, by way of example, withreference to the accompanying drawings, in which:

FIG. 1 illustrates an exemplary arrangement of a vehicle equipped with acamera and a radar imaging system in accordance with the disclosure;

FIG. 2 illustrates a known method to determine a clear path forautonomous driving in accordance with the disclosure;

FIG. 3 illustrates an exemplary method to determine a clear pathutilizing a likelihood analysis of an image in accordance with thedisclosure;

FIG. 4 illustrates an exemplary method to analyze an image in accordancewith the disclosure;

FIG. 5 illustrates an exemplary method to define a classification errorby tuning a single threshold in accordance with the disclosure;

FIGS. 6A, 6B, and 6C illustrate an exemplary determination of an imagedifference by calculating an absolute image intensity difference inaccordance with the disclosure;

FIG. 7 illustrates an exemplary method to classify a feature as aportion of a clear path and as a detected object at the same time as amethod of image analysis in accordance with the disclosure;

FIG. 8 further illustrates an exemplary method to classify a feature asa portion of a clear path and as a detected object at the same time as amethod of image analysis in accordance with the disclosure;

FIG. 9 is a flowchart for an exemplary patch-based method for detectinga clear path in accordance with the disclosure;

FIG. 10 is a flowchart for a texture-rich pixel-based method fordetecting a clear path, in accordance with the disclosure;

FIG. 11 is a flowchart for a textureless pixel-based method fordetecting a clear path, in accordance with the disclosure;

FIG. 12 shows an exemplary image used to illustrate a vehicle detectionanalysis, in accordance with the disclosure;

FIG. 13 shows an exemplary image used to illustrate a construction areadetection analysis, in accordance with the disclosure;

FIG. 14 is a flowchart for an exemplary method for augmenting the clearpath detection analysis with detection of objects around the vehicle, inaccordance with the disclosure;

FIG. 15 shows a schematic diagram of an exemplary vehicle system whichhas been constructed with the target tracking system, in accordance withthe present disclosure;

FIG. 16 shows a control system for information flow utilized in creatinga track list, in accordance with the present disclosure;

FIG. 17 depicts an exemplary data fusion process, in accordance with thepresent disclosure;

FIG. 18 depicts an exemplary dataflow enabling joint tracking and sensorregistration, in accordance with the present disclosure;

FIG. 19 schematically illustrates an exemplary system whereby sensorinputs are fused into object tracks useful in a collision preparationsystem, in accordance with the present disclosure;

FIG. 20 schematically illustrates an exemplary image fusion module, inaccordance with the present disclosure;

FIG. 21 schematically depicts an exemplary bank of Kalman filtersoperating to estimate position and velocity of a group objects,accordance with the present disclosure;

FIG. 22 illustrates exemplary range data overlaid onto a correspondingimage plane, useful in system-internal analyses of various targetobjects, in accordance with the present disclosure; and

FIG. 23 graphically depicts an exemplary iterative analysis, inaccordance with the present disclosure.

DETAILED DESCRIPTION

Referring now to the drawings, wherein the showings are for the purposeof illustrating certain exemplary embodiments only and not for thepurpose of limiting the same, FIG. 1 illustrates an exemplaryarrangement of camera 110 located on the front of a vehicle 100 andpointed toward the ground in front of the vehicle 100 in accordance withthe disclosure. Camera 110 is in communication with processing module120 containing logic to process inputs from camera 110. The vehicle 100may also be equipped with a radar imaging system 130, which, whenpresent, is also in communication with processing module 120. It shouldbe appreciated by those having ordinary skill in the art that thevehicle 100 could utilize a number of methods to identify roadconditions in addition or in the alternative to the use of camera 110and the radar imaging system 130, including GPS information, informationfrom other vehicles in communication with the vehicle 100, historicaldata concerning the particular roadway, biometric information such assystems reading the visual focus of the driver, a radar imaging system,or other similar systems. The disclosure herein can be applied tovarious device arrangements and is therefore not limited thereby.

The camera 110 is a device well known in the art capable of translatingvisual inputs in the form of light, infrared, or other electro-magnetic(EM) radiation into a data format readily capable of analysis, e.g., adigital, pixelated image. In one embodiment, the camera 110 uses acharge coupled device (CCD) sensor to generate images indicating afield-of-view. Preferably, the camera 110 is configured for continuousimage generation, e.g., 30 images generated per second. Images generatedby the camera 110 may be stored in memory within the camera 110 ortransferred to the processing module 120 for storage and/or analysis.Preferably, each image generated by the camera 110 is a two-dimensionalimage of known pixel dimensions comprising a plurality of identifiablepixels. The plurality of identifiable pixels may be stored and analyzedusing an array. Each pixel may be represented in the array as a set ofbits or a plurality of sets of bits wherein the bits correspond to acolor on a predetermined palette or color map. Each pixel may beexpressed as a function of a plurality of color intensity values such asin a red-green-blue (RGB) color model or a cyan-magenta-yellow-key(CMYK) color model. Preferably, each pixel comprises a plurality of setsof bits wherein each set of bits corresponds to a color intensity and acolor intensity value e.g., a first set of bits corresponds to a redcolor intensity value, a second set of bits corresponds to a green colorintensity value, and a third set of bits corresponds to blue colorintensity value on the RGB color model.

The radar imaging device 130 is a device well known in the artincorporating a transmitter capable of emitting radio waves or other EMradiation, a receiver device capable of sensing the emitted wavesreflected back to the receiver from objects in front of the transmitter,and means to transfer the sensed waves into a data format capable ofanalysis, indicating for example range and angle from the objects offwhich the waves reflected. Alternatively, the radar imaging device 130may be replaced or supplemented with a light detection and ranging(LIDAR) system configured to transmit and receive optical energy. Thereceived optical energy may be used to determine object geometricdimensions and/or geometrical proximity to the vehicle 100. It will benoted that radar imaging device 130 is optional and unnecessary toperform many of the methods disclosed herein, wherein processing ofvisual images is capable of accomplishing clear path detection. The term“clear path” as used herein is to be given its ordinary and customarymeaning to a person of ordinary skill in the art (and it is not to belimited to a special or customized meaning), and refers withoutlimitation to a path free of objects exceeding a threshold.

The processing module 120 is illustrated in FIG. 1, and described hereinas a discrete element. Such illustration is for ease of description andit should be recognized that the functions performed by this element maybe combined in one or more devices, e.g., implemented in software,hardware, and/or application-specific integrated circuitry. Theprocessing module 120 can be a general-purpose digital computercomprising a microprocessor or central processing unit, storage mediumscomprising non-volatile memory including read only memory andelectrically programmable read only memory, random access memory, a highspeed clock, analog to digital and digital to analog circuitry, andinput/output circuitry and devices and appropriate signal conditioningand buffer circuitry. In the alternative, processing module 120 can be adigital signal processing (DSP) unit, such as a customized integratedcircuit such as a field programmable gate array. The processing module120 has a set of processing algorithms, comprising resident programinstructions and calibrations stored in the non-volatile memory andexecuted to provide desired functions. The algorithms are preferablyexecuted during preset loop cycles. Algorithms are executed by thecentral processing unit and are operable to monitor inputs from theaforementioned sensing devices and execute control and diagnosticroutines to control operation of the actuators, using presetcalibrations. Loop cycles may be executed at regular intervals, forexample each 3.125, 6.25, 12.5, 25 and 100 milliseconds during ongoingengine and vehicle operation. Alternatively, algorithms may be executedin response to occurrence of an event.

The processing module 120 executes algorithmic code stored therein tomonitor related equipment such as camera 110 and radar imaging system130 and execute commands or data transfers as indicated by analysisperformed within the processing module. Processing module 120 mayinclude algorithms and mechanisms to actuate autonomous driving controlby means known in the art and not described herein, or processing module120 may simply provide information to a separate autonomous drivingsystem. Processing module 120 is adapted to receive input signals fromother systems and the operator as necessary depending upon the exactembodiment utilized in conjunction with the control module.

FIG. 2 illustrates a known method to determine a clear path forautonomous driving in accordance with the disclosure. Image 10 isgenerated corresponding to the roadway in front of vehicle 100. Throughone of various methods, objects 40A, 40B, and 40C are identified withinimage 10, and each object is categorized and classified according tofiltering and trained object behaviors. Separate treatment of eachobject can be computationally intensive, and requires expensive andbulky equipment to handle the computational load. An algorithm processesall available information regarding the roadway and objects 40 toestimate a clear path available to vehicle 100. Determination of theclear path depends upon the particular classifications and behaviors ofthe identified objects 40.

FIG. 3 illustrates an exemplary method to determine a clear path forautonomous or semi-autonomous driving in accordance with the disclosure.Image 10 is depicted including ground 20, horizon 30, and objects 40.Image 10 is collected by camera 110 and represents the road environmentin front of vehicle 100. Ground 20 represents the zone of all availablepaths open to travel without regard to any potential objects. The methodof FIG. 3 that determines a clear path upon ground 20 starts bypresuming all of ground 20 is clear, and then utilizes available data todisqualify portions of ground 20 as not clear. In contrast to the methodof FIG. 2 which classifies every object 40, the method of FIG. 3 insteadanalyzes ground 20 and seeks to define a clear path confidencelikelihood from available data that some detectable anomaly which mayrepresent object 40 limits or makes not clear that portion of ground 20.This focus upon ground 20 instead of objects 40 avoids the complexcomputational tasks associated with managing the detection of theobjects. Individual classification and tracking of individual objects isunnecessary, as individual objects 40 are simply grouped together as apart of the overall uniform limitation upon ground 20. Ground 20,described above as all paths open to travel without discrimination,minus limits placed on ground 20 by areas found to be not clear, defineclear path 50, depicted in FIG. 3 as the area within the dotted lines,or an area with some threshold confidence likelihood of being open fortravel of vehicle 100.

Object 40 that creates not clear limitations upon ground 20 can takemany forms. For example, an object 40 can represent a discrete objectsuch as a parked car, a pedestrian, or a road obstacle, or object 40 canalso represent a less discreet change to surface patterns indicating anedge to a road, such as a road-side curb, a grass line, or watercovering the roadway. Object 40 can also include an absence of flat roadassociated with ground 20, for instance, as might be detected with alarge hole in the road. Object 40 can additionally include an indicatorwithout any definable change in height from the road, but with distinctclear path implications for that segment of road, such as a paintpattern on the roadway indicative of a lane marker. The method disclosedherein, by not seeking to identify object 40 but by taking visual cuesfrom ground 20 and anything in proximity to the ground in image 10,evaluates a clear path confidence likelihood of clear versus not clearand adjusts the control of vehicle 100 for the presence of any object40.

Numerous methods for automated analysis of two-dimensional (2D) imagesare possible. Analysis of image 10 is performed by an algorithm withinprocessing module 120. FIG. 4 illustrates one exemplary method which maybe applied to analyze image 10 in accordance with the disclosure. Thismethod subdivides image 10 and identifies a sub-image or patch 60 ofground 20 for analysis, extracts features or analyzes the availablevisual information from patch 60 to identify any interesting ordistinguishing features within the patch, and classifies the patchaccording to a confidence likelihood of being a clear path according toanalysis of the features. Patches with greater than a certain thresholdof likeliness are classified as clear, and a compilation of patches canbe used to assemble a clear path within the image.

Patch 60, as a sub-image of image 10, can be identified through anyknown means, such as random search or swarm search of image 10.Alternatively, information regarding the presence of an object 40available from some other source of information, such as radar imagingsystem 130, can be used to identify a patch to analyze the portion ofimage 10 which should describe object 40. Image 10 may require manypatches 60 to analyze the whole image. In addition, multiple overlayingpatches or patches of different size could be used to fully analyze aregion of image 10 containing information of interest. For instance, asmall patch 60 might be used to analyze a small dot on the road;however, a large patch 60 might be required to analyze a series of dotswhich in isolation might seem uninteresting, but in context of theentire series, could indicate an object 40 of interest. In addition, theresolution of patches applied to a particular area may be modulatedbased upon information available, for instance, with more patches beingapplied to a region of image 10 wherein an object 40 is thought toexist. Many schemes or strategies can be utilized to define patches 60for analysis, and the disclosure is not intended to be limited to thespecific embodiments described herein.

Once a patch 60 has been identified for analysis, processing module 120processes the patch by applying known feature identification algorithmsto the patch. Additionally, processing module 120 may perform analysisof the location of the patch in context to the location of the vehicle.Feature identification algorithms search available visual informationfor characteristic patterns in the image associated with an objectincluding features defined by line orientation, line location, color,corner characteristics, other visual attributes, and learned attributes.Feature identification algorithms may be applied to sequential images toidentify changes corresponding to vehicle motion, wherein changes notassociated with ground movement may be identified not clear path.Learned attributes may be learned by machine learning algorithms withinthe vehicle, but are most frequently programmed offline and may bedeveloped experimentally, empirically, predictively, through modeling orother techniques adequate to accurately train distinguishing attributes.

Once features in patch 60 have been extracted, the patch is classifiedon the basis of the features to determine the confidence likelihood thatthe patch is a clear path. Likelihood analysis is a process known in theart by which a likelihood value or a confidence is developed that aparticular condition exists. Applied to the present disclosure,classification includes likelihood analysis to determine whether thepatch represents a clear path or if ground 20 in this patch is limitedby an object 40. Classification is performed in an exemplary embodimentby application of classifiers or algorithms trained with a database ofexemplary road conditions and interactions with detected objects. Theseclassifiers allow processing module 120 to develop a fractional clearpath likelihood value for patch 60, quantifying a confidence betweenzero and one that the features identified within the patch do notindicate a limiting object 40 which would inhibit free travel of vehicle100. A threshold confidence can be set, defining the clear pathlikelihood required to define the patch as a clear path, for instance bythe following logic:Confidence=ClearPathLikelihood(i)If _Confidence>0.5, then _patch=clearpath  (1)In this particular exemplary embodiment, a confidence of 50% or 0.5 isselected as the threshold confidence. This number can be developedexperimentally, empirically, predictively, through modeling or othertechniques adequate to accurately evaluate patches for clear pathcharacteristics.

The likelihood analysis, as mentioned above, may be performed in oneexemplary embodiment by application of trained classifiers to featuresextracted from a patch. One method analyzes the features a-priori usinga training set of images. In this training stage, distinguishingfeatures are selected from a raw feature set, the distinguishingfeatures being defined by methods known in the art, such as Haarwavelet, Gabor wavelet, and Leung-and-Malik filter bank. In addition, 2Dimage location information based on each feature's minimalclassification errors, calculated as the sum of false acceptance rate(FAR) and false rejection rate (FRR), may be utilized by tuning a singlethreshold as illustrated in FIG. 5. This classification error can bedescribed through the following expression:ClassificationError(i)=FAR_(i)+FRR_(i)  (2)Information from the trained classifiers is used to classify or weightthe feature as indicating a clear path or not clear path, the particularclassification depending upon the strength of comparisons to the traineddata. Classification of the feature, if the feature is the only featurewithin the patch, may be directly applied to the patch. Classificationof a patch with multiple features identified may take many forms,including the patch being defined by the included feature mostindicative of the patch being not clear or the patch being defined by aweighted sum of all of the features included therein.

The above method can be utilized to examine an individual image 10 andestimate a clear path 50 based upon visual information contained withinimage 10. This method may be repeated at some interval as the vehicletravels down the road to take new information into account and extendthe formulated clear path to some range in front of the vehicle's newposition. Selection of the interval must update image 10 with enoughfrequency to accurately supply vehicle 100 with a clear path in which todrive. However, the interval can also be selected to some minimum valueto adequately control the vehicle but also not to unduly burden thecomputational load placed upon processing module 120.

Clear path detection can be accomplished through a single image 10 asdescribed above. However, processing speed and accuracy can be improvedwith the addition of a second image taken in close time proximity to theoriginal image, such as sequential images from a streaming video clip. Asecond image allows direct comparison to the first and provides forupdated information regarding progression of the vehicle and movement ofdetected objects. Also, the change of perspective of camera 110 allowsfor different analysis of features from the first image: a feature thatmay not have shown up clearly or was indistinct in the first image maydisplay at a different camera angle, stand out more distinctly, or mayhave moved since the first image, allowing the classification algorithman additional opportunity to define the feature.

Processing of a second image in relation to the original image 10 can beperformed by calculating an image difference. If the image difference ofa point of interest, such as a feature identified by radar, is not zero,then the point can be identified as embodying new information. Pointswhere the image difference does equal zero can be eliminated fromanalysis and computation resources may be conserved. Methods todetermine image difference include absolute image intensity differenceand vehicle-motion compensated image difference.

Determining an image difference by calculating an absolute imageintensity difference can be used to gather information between twoimages. One method of absolute image intensity difference includesdetermining equivalent image characteristics between the original imageand the second image in order to compensate for movement in the vehiclebetween the images, overlaying the images, and noting any significantchange in intensity between the images. A comparison between the imagesindicating a change in image intensity in a certain area contains newinformation. Areas or patches displaying no change in intensity can bede-emphasized in analysis, whereas areas displaying clear changes inintensity can be focused upon, utilizing aforementioned methods toanalyze patches on either or both captured images.

FIGS. 6A, 6B, and 6C illustrate an exemplary determination of an imagedifference by calculating an absolute image intensity difference inaccordance with the disclosure. FIG. 6A depicts an original image. FIG.6B depicts a second image with changes from the original image. Inparticular the depicted circular shape has shifted to the left. Acomparison of the two images as illustrated in FIG. 6C, an outputrepresenting the result of an absolute image intensity differencecomparison, identifies one region having gotten darker from the firstimage to the second image and another region having gotten lighter fromthe first image to the second image. Such a method can be described asdifferencing. Analysis of the comparison yields information that somechange as a result of movement or change of perspective is likelyavailable in that region of the images. In this way, absolute imageintensity difference can be used to analyze a pair of sequential imagesto identify a potentially not clear path.

Likewise, determining an image difference by calculating avehicle-motion compensated image difference can be used to gatherinformation between two images. Many methods to calculate avehicle-motion compensated image difference are known. One exemplarymethod of vehicle-motion compensated image difference includes analyzinga potential object as both a stationary portion of a clear path and adetected object at the same time. Likelihood analysis is performed onfeatures identified corresponding to the potential object from bothclassifications at the same time, and the classifications may becompared, for example, through the following logic:Confidence(i)=ClearPathLikelihood(i)−DetectedObjectLikelihood(i)If _Confidence>0, then _patch=clearpath  (3)In this exemplary comparison, if confidence(i) is greater than zero,then the patch containing the feature is classified as a clear path. Ifconfidence(i) equals or is less than zero, then the patch containing thefeature is classified as not a clear path or limited. However, differentvalues may be selected for the confidence level to classify the patch asa clear path. For example, testing may show that false positives aremore likely than false negatives, so some factor or offset can beintroduced.

FIG. 7 illustrates one method to classify a feature as a portion of aclear path and as a detected object at the same time as described abovein accordance with the disclosure. Image 10 includes object 40,trapezoidal projection 70, and rectangular projection 80. This methodutilizes an assumption projecting object 40 as a flat object on theground within projection 70 to test the classification of the feature asa portion of a clear path. The method also utilized an assumptionprojecting object 40 as a vertical object within rectangular projection80 to test the classification of the feature as a detected object. FIG.8 illustrates comparisons made in data collected between the two imagesto evaluate the nature of object 40 in accordance with the disclosure.Camera 110 at time t₁ observes and captures data from object 40 in theform of a first image. If object 40 is an actual detected object, theprofile observed by camera 110 of object 40 at time t₁ will correspondto point 90A. If object 40 is a flat object in the same plane as ground20, then the profile observed by camera 110 of object 40 at time t₁ willcorrespond to point 90B. Between times t₁ and t₂, camera 110 travelssome distance. A second image is captured at time t2, and informationregarding object 40 can be tested by applying an algorithm looking atvisible attributes of the object in the second image in comparison tothe first image. If object 40 is an actual detected object, extendingupward from ground 20, then the profile of object 40 at time t₂ will beobserved at point 90C. If object 40 is a flat object in the same planeas ground 20, then the profile of object 40 at time t2 will be observedat point 90B. The comparison derived through vehicle-motion compensatedimage difference can directly assign a confidence by application ofclassifiers based on the observations of points 90, or the comparisonmay simply point to the area displaying change as a point of interest.Testing of the object against both classifications, as a flat object andas an actual detected object, allows either the area including object 40to be identified for further analysis through analysis of a patch asdescribed above or direct development of a clear path likelihood and adetected object likelihood for comparison, for example as in logicexpression (3) above.

Information available from analysis of the second image can additionallybe improved by integration of information regarding movement of thevehicle, such as speed and yaw-rate. Information regarding vehiclemotion is available from a number of sources, including the vehiclespeedometer, vehicle dynamic sensors or wheel speed sensors, anti-lockbraking mechanisms, and GPS location systems. Algorithms may utilizethis vehicle movement information, for example, in conjunction with theprojections described in FIGS. 7 and 8 to project angles which shouldexist in a feature laying flat on the ground in the second image basedupon data from the first image and the measured movement of the vehiclebetween the images.

The number of images utilized for comparison need not be limited to two.Multiple image analysis can be performed at multiple iterations, with anobject being tracked and compared over a number of cycles. As mentionedabove, computational efficiency can be gained by utilizing imagedifference analysis to identify points of interest and eliminating areaswith zero difference from subsequent analyses. Such efficiencies can beused in multiple iterations, for example, saying that only points ofinterest identified between a first and second image will be analyzed inthe third and fourth images taken. At some point, a fresh set of imageswill need to be compared to ensure that none of the areas showing zerodifference have had any change, for example a moving object impingingupon a previously identified clear path. The utilization of imagedifference analyses and of focused analyses, eliminating areasidentified with zero change, will vary from application to applicationand may vary between different operating conditions, such as vehiclespeed or perceived operating environment. The particular utilization ofimage difference analyses and of focused analyses can take manydifferent embodiments, and the disclosure is not intended to be limitedto the specific embodiments described herein.

Multiple methods are herein disclosed for using the clear path detectionanalysis to identify a clear path including patch-based methods andpixel-based methods. The methods herein are preferably executed in theprocessing module 120, but may be combined in one or more devices, e.g.,implemented in software, hardware, and/or application-specificintegrated circuitry. Patch-based methods are described above and hereinwith reference to FIG. 9. Pixel-based methods are described herein withreference to FIGS. 11 and 12. It will be appreciated that pixel-basedmethods make decisions based upon pixel or a group of pixels. Thisdecision for the component patch being analyzed can be based upon pixelscontained within the patch, for example, with the patch being determinedto be unclear if any or some minimum number of pixels within the patchare determined to be not clear. Exemplary pixel-based methods includetextureless and texture-rich methods. Texture-rich methods analyzepixelated features in an image for distinct interest points based upon acontextual view within the image. The interest points are mappedcorresponding to a field-of-view in from on the vehicle 100 and a clearpath is determined based upon topography of the interest points withinthe map. Textureless methods filter from an image non-conforming regionsof the image as not belonging to a planar, consistent road surface,remaining pixels correspond to the clear path. As described hereinabove,patch-based methods are computationally, relatively fast, whilepixel-based methods are computationally, relatively slow.

FIG. 9 illustrates an exemplary patch-based method 200 wherein inputfrom a camera is analyzed to determine a clear path likelihood inaccordance with the disclosure. The patch-based method 200 isillustrated in FIG. 9, and described herein as comprising discreteelements. Such illustration is for ease of description and it should berecognized that the functions performed by these elements may becombined in one or more devices, e.g., implemented in software,hardware, and/or application-specific integrated circuitry. For example,the patch-based method 200 may be executed as one or more algorithms inthe processing module 120.

During operation, the camera 110 generates an image for analysis in theprocessing module 120 (202). The processing module 120 identifiespatches in the image and selects a patch for analysis (204). Multiplemethods for defining a plurality of patches sufficient to adequatelyanalyze the image are contemplated by this disclosure. As describedabove, patches may be defined according to random search or swarm searchmethods. Alternatively, information from some other source ofinformation, such as a radar imaging system, can be used to define apatch to analyze the portion of the image. In addition, multipleoverlaying patches can be utilized based upon the perceived area ofinterest upon the image to be analyzed. Additionally, methods can beutilized to define patches according to anticipated road surface infront of the vehicle and resulting clear path patterns, for example,including a fixed-grid patch method, sectioning off some or all of theimage according to a regular patch pattern, and a perspective patchmethod, similar to the fixed-grid patch method except that the patchsizes and/or shapes are modulated based upon a perspective view of theroad and consideration for an amount of road surface contained withineach of the patches. Such an anticipated road surface in front of thevehicle can be adjudged by an initial review of the input image, forexample, utilizing clearly visible lines that could define laneboundaries as an initial reference to define a patch pattern. In anotherexample, a patch pattern from an immediately previous iteration of theclear path method could be utilized and slowly modulated through theiterations, based upon the iteratively defined clear path and otherindications that can be identified within the analyzed images.

The fixed-grid patch method identifies and divides the image into aplurality of patches based upon an area of interest and applies afixed-grid pattern to the area of interest. The fixed-grid patternsections substantially the entire area of interest in patches. The areaof interest preferably encompasses the field-of-view immediately infront of the vehicle, however the area may be defined to encompassesmore narrow fields-of-view. In one embodiment, the area of interestencompasses the field-of-view immediately in front of the vehicle anddelimited by a horizon line or a vanishing point perceived for theroadway. The fixed-grid patch method can include patch tracking,identification, and analysis via a matrix, wherein each patch may beidentified by a number of a number series.

The fixed-grid patch may be applied to the image using a number ofmethods. A first method includes applying the fixed-grid patch tosequential images using equal image coordinates. A second methodincludes applying the fixed-grid patch to the image using an identifiedinterest point on the image, e.g., the horizon line. A third methodincludes applying the fixed-grid patch to the image compensating forvehicle movement, e.g., vehicle yaw.

The perspective patch method identifies and divides the image into aplurality of patches based upon field-of-view coordinates rather thanimage coordinates. An area of interest is identified as describedhereinabove, wherein a perspective patch pattern is applied to the areaof interest based upon estimated field-of-view coordinates, allocatingpatch size relative to an approximate area of ground covered by eachpatch. Such perspective patch definitions allow for a more detailedreview of patches describing ground closer to the vehicle and lesswasteful review of patches describing ground more distant from thevehicle. Patches can be aligned to the perceived lanes of travel uponthe roadway, for example, as defined by lane markers and/or curbsides.Such patch definitions will frequently be trapezoidal, with the parallelsides of the trapezoids being parallel to the horizon or horizontal axisof the vehicle, and with the remaining sides of the trapezoids beingdependent upon the defined road surface in the image view. Such a patchalignment is efficient to defining the road surface. However,computation of the grid of patches and the analysis of the patches iscomplicated by the many different shapes. The patches can instead benormalized to rectangle (or square) shapes, still parallel to thehorizon or horizontal axis of the vehicle. Such rectangular patches arein ways less efficient in covering the road surface, for example, withportions of the curbside overlapping patches including actually clearroadway, but sufficient resolution of the patches and improved thecomputational efficiency can make such normalized perspective patchesbeneficial.

A filter or a set of filters may be applied to the selected patch (206),e.g., a lighting normalization filter. An exemplary normalization filterchanges the range of pixel intensity values within the patch, therebybringing the patch into a range that is more suitable for machineprocesses. For example, normalizing each pixel value to a zero mean andunit variance enhances the image contrast, specifically in a lowlighting environments or when contrast is poor due to glare. A number ofexemplary filters and filtering methods useful for image analysis areenvisioned, and the disclosure is not intended to be limited to theparticular exemplary embodiments described herein.

After filtering, feature extraction is executed on the selected patchusing feature extraction algorithms (208). Feature extraction algorithmsmay analyze the selected patch for predetermined features, e.g., edges,corners, and blobs, and/or shapes, e.g., circles, ellipses, and lines.It will be appreciated that some features have meaning and others donot, and a process of feature selection can be utilized to identify agroup of best features for analysis. A classifier training algorithmanalyzes each feature and assigns a likelihood value (210). As mentionedabove, classifiers or logic used in developing likelihood values areinitially trained offline. Training may optionally be continued in thevehicle based upon fuzzy logic, neural networks, or other learningmechanisms known in the art. These trained classifiers perform alikelihood analysis (212) upon the features extracted, and a likelihoodvalue for the patch is determined. This likelihood value expresses aconfidence that the selected patch is clear.

The likelihood analysis may be augmented using spatially and/ortemporally related patches to evaluate identified features duringvehicle operation. When the feature extraction algorithm has identifieda feature, the processing module 120 may spatially analyze theidentified feature for consistency among adjacent or nearby patches. Afeature identified in one patch may be compared to surrounding patchesto determine if it is an aberration or consistent with the surroundingpatches. A feature that is consistent with surrounding patches may beassigned a similar likelihood value to the surrounding patches, whereasa feature not consistent with surrounding patches can be assigned adifferent likelihood value. Similarly to the spatial analysis, when thefeature extraction algorithm has identified a feature, the processingmodule 120 may temporally analyze the identified feature for consistencyamong temporally related patches, compensating for vehicle motion. Forexample, a feature analyzed in several frames and determined to have ahigh likelihood value through the image frames can be temporallycompared to the same feature receiving a low likelihood value in a laterframe. If the temporal analysis of the feature reveals new information,such as movement of the feature with respect to the roadway or increasedperspective revealing the feature to be indicative of an object situatedupon the road surface, then the lower likelihood value for the featurecan be confirmed. If no new information is available, then the lowerlikelihood value for the feature in the present frame can be suspendedas not necessarily indicating a not clear path. Subsequent frames can beanalyzed similarly to establish the feature as significant or not.Similarly, according to methods described herein, the questionablefeature can be analyzed with increased computational emphasis, either inthe present image or in subsequent images.

The processing module 120 compares the likelihood value to a thresholdlikelihood value (214). If the likelihood value is greater than thethreshold value, then the patch is identified as a clear path (218). Ifthe likelihood value is not greater than the threshold value, then thepatch is identified as a not clear path (216).

As described above, the patch-based method 200 may be repeated orreiterated in a number of ways, with the same image being analyzedrepeatedly with the selection and analysis of different patches, and anidentified patch can be tracked and analyzed for change over a number ofsequential images.

FIG. 10 illustrates an exemplary texture-rich pixel-based method 300wherein input from a camera is analyzed to determine a clear pathlikelihood in accordance with the disclosure. The texture-richpixel-based method 300 is illustrated in FIG. 10, and described hereinas comprising discrete elements. Such illustration is for ease ofdescription and it should be recognized that the functions performed bythese elements may be combined in one or more devices, e.g., implementedin software, hardware, and/or application-specific integrated circuitry.For example, the pixel-based method 300 may be executed as one or morealgorithms in the processing module 120.

During operation, the camera 110 generates an image for analysis in theprocessing module 120 (302). The processing module 120 analyzes theimage for interest points, for example, examining pixel color intensityas described hereinabove and comparing the pixel or a group of pixelswith surrounding pixels. An interest point is an identifiable pixel onan image and is associated with a set of visual information, i.e.,texture-rich features, and is associated with objects located in thefield-of-view. Through methods known in the art, e.g., a scale-invariantfeature transform (SIFT), methods employing corner detection or othershape detection, or a Sobel filter, interest points can be identifiedand utilized to accomplish methods described herein (306). In oneembodiment, redundant interest points, e.g., multiple interest pointscorresponding to the same feature, are removed for computationalefficiency considerations.

Once the interest points are identified, the processing module 120compares sequential images when the vehicle is in motion to matchinterest points from each image to corresponding points in thesequential images which correspond to the same points in thefield-of-view, where possible (310). Matching includes usingcorrespondence matching programming, e.g., a scale-invariant featuretransform (SIFT) feature matching programming and optical flowprogramming, and may further include locating corresponding pointsthrough template matching, taking into account movement of the hostvehicle, and making a best estimate whether two points represent thesame object or feature visible in the field-of-view. Template matchingmay be determined using one of multiple methods, including one ofseveral known template matching programming methods to find thecorresponding interest points, e.g., Lucas-Kanade or Horn-Schunck. Theresulting matched point pairs correspond to a same feature located onboth images wherein the same feature is associated with a same object inthe field-of-view. While interest points can be matched, not all matchedcorresponding point pairs represent high quality corresponding pointpairs that allow the identification of their three-dimensional positionsin the field-of-view for classifications as a clear path for the vehicleto travel through.

The processing module 120 filters the matched corresponding point pairsin order to identify high quality corresponding point pairs that can beutilized for three-dimensional position identification with highconfidence (314). Preferential matched point pairs may be selected basedupon quality control criteria, e.g., distance between points, distancefrom image boundary, and color similarities between respectiveneighboring pixels. Selection of criteria to judge matched pairs canadditionally be made based upon conditions such as light level presentoutside the vehicle, weather, speed of the vehicle, and any other factoraffecting an ability to judge matched pairs or an urgency to quickly andaccurately define a clear path.

The high quality corresponding point pairs are analyzed to determinethree-dimensional positions of objects in the field-of-view representedby the corresponding point pairs (318). It will be appreciated thatcorresponding points at different heights as compared to ground levelwill move differently between sequential images. Analyzing movement ofthe interest points between sequential images can yield estimatedthree-dimensional coordinates of the interest points. Object positioncan be determined based upon the high quality corresponding point pairs,sample time between sequential images, and vehicular data such asvehicle speed, and vehicle yaw rate. These methods of triangulation canyield a position of the object in a horizontal plane and a height of theobject in relation to a ground level.

The determined object positions are utilized to map object positions infront of the host vehicle including an estimated topography of thefield-of-view (322). The topography may be estimated by assigningpredetermined spatial dimensions around the object. Preferably, thepredetermined spatial dimensions diminish with respect to height at apredetermined rate with respect to distance from the object. Using themap and estimated topography the processing module 120 can determine aclear path in front of the host vehicle (326).

The above method utilizes sequential images to establish a map of objectpositions and vertical heights in front of the vehicle, such that aclear path can be defined. It will be appreciated that in any two givenimages, a particular object might not be classified as including twohigh quality interest points sufficient to be mapped in that particularanalysis. However, the above analysis occurs multiple times per secondof vehicle travel. As the vehicle travels forward through the clearpath, different perspectives upon an object will be gained and a largenumber of images will be analyzed. Travel over a path and analysis ofthe multiple iterative images through that path build a confidencethrough the analyses that no object contradicting the clear path existsin the identified clear path.

FIG. 11 illustrates an exemplary textureless pixel-based method 400wherein input from a camera is analyzed to determine a clear pathlikelihood in accordance with the disclosure. The texturelesspixel-based method 400 is illustrated in FIG. 11, and described hereinas comprising discrete elements. Such illustration is for ease ofdescription and it should be recognized that the functions performed bythese elements may be combined in one or more devices, e.g., implementedin software, hardware, and/or application-specific integrated circuitry.For example, the textureless pixel-based method 400 may be executed asone or more algorithms in the processing module 120.

During operation, the camera 110 generates an image for analysis in theprocessing module 120 (453). The processing module 120 analyzes theimage using multiple filtering methods to identify and removenon-conforming pixels from the image. Remaining pixels indicate apotential clear path for the vehicle to travel. FIG. 11 shows thetextureless pixel-based method 400 including four exemplary filteringmethods to identify and remove non-conforming pixels from the image. Atextureless method could be used with some portion of the four exemplarymethods and/or can include unnamed but similar methods to process theimage.

A first exemplary filtering method removes pixels above a horizon orvanishing point, including sky and other vertical features that cannotbe part of a road surface (456). The term “vanishing point” as usedherein is a broad term, and is to be given its ordinary and customarymeaning to one ordinarily skilled in the art, and refers to an infinitefar point on the horizon that is intersected by multiple parallel lineson the ground in the view. Identifying a road surface creating a clearpath on which to drive is necessarily below the vanishing point orhorizon line. Filtering images to only analyze an area below the horizonline helps to clarify the pixels being analyzed to identify a roadsurface from irrelevant pixels. As one skilled in the art appreciates,there are many known methods for determining a vanishing point andcorresponding horizon line.

A second exemplary filtering method applies a filter based upon variancein pixel color intensity, based upon a premise that a road surface willinclude a large surface with a visual intensity common across thesurface (459). Pixels are removed from the image when an associatedpixel color intensity varies greater than a predetermined threshold. Forpixels associated with multiple colors, a pixel color intensity of anyparticular color that varies greater than the predetermined thresholdmay be removed from the image. The predetermined threshold may beupdated based upon historical color intensity of pixels identified asclear.

A third exemplary filtering method applies a filter based upondifferencing sequential images, allowing analysis of changes between theimages (462). Pixels associated with a pixel color intensity thatchanges greater than a predetermined threshold from the sequentialimages may be removed from the image. In one embodiment, the adjustmentsare made to the images based upon motion of the vehicle so that pixelsare differenced and compared as if all pixels correspond to points at aground level. Known triangulation methods may be used for determiningmotion adjustments to the images. By using an assumption that allobjects in the image are at ground level, non ground level points may beidentified by changes not consistent with pixels associated with theground level. For example, pixels above ground level may move fasterbetween sequential images than would be expected, and this movement maybe identified by examining the difference between pixel colorintensities between sequential images.

A fourth exemplary filtering method applies a filter based uponidentifying pixels representing edges or transitions in the visual data(465). To create the fourth filtered image, the processing module 120extracts pixels from the image based upon color intensity values thatcorrespond to edges using one of several known edge detection filters,e.g., a Sobel filter. The edge detection filter is preferably executedin the processing module 120, but may be combined in one or moredevices, e.g., implemented in software, hardware, and/orapplication-specific integrated circuitry. In one embodiment, each pixelis analyzed using a Sobel operator. The Sobel operator computes agradient vector of color intensity at each pixel resulting in adirection of the largest possible increase from light to dark and therate of change in that direction. Points corresponding to a rate ofchange exceeding a threshold and corresponding to gradient vectors atnearby pixels indicate edges and are included in the fourth filteredimage. Those pixels are included in the image while the others areremoved.

Applying the various methods in parallel, the results can be the fusedinto a single map of the image (468). Fusion includes pixels identifiedin each of the various filtering methods. Pixels on the fused clear pathmap correspond to desirable driving locations in the field-of-view.Locations on the fused clear path map without pixels correspond toundesirable driving locations in the field-of-view. The processingmodule 120 may analyze the map for visual data indicative of a clearpath of travel (471).

The textureless pixel-based method 400, described in FIG. 11, appliesvarious methods to images in parallel to identify features in a viewrelevant to defining a clear path. However, it will be appreciated thatthese methods need not be performed in parallel, but rather the methodscan be used to process images sequentially in steps or selectively toidentify features in a view relevant to defining a clear path.

An addition pixel-based clear path detection method includes applying afused texture-rich and textureless method.

The texture-rich and textureless methods can be fused in a number ofways. An image with identified points and determined heights identifiedwith texture-rich methods can be overlaid with a filtered imagegenerated by textureless methods, and agreement of the two methods canbe used to define a clear path through the overlaid image. In analternative method to fuse the two schemes, the data from each of thetwo schemes can be used to project information upon a programmedoverhead map of an area in front of the vehicle, and this overhead mapincluding data gained from analysis of the two schemes can includebuilding confidence indications for regions of the map. In analternative method to fuse the two schemes, one scheme can be utilizedas a primary or dominant scheme, and the second scheme can be utilizedor activated to analyze regions in the view identified as ambiguous orunclear. In any method to fuse the two schemes, strengths of oneprocessing scheme can be used to reduce weaknesses of the otherprocessing scheme. If both schemes concur that the path is clear, thenthe processing module employing the schemes may determine with increasedconfidence that the path is desirable for the vehicle to traverse. Anumber of methods to fuse the identified schemes are envisioned, and thedisclosure is not intended to be limited to the particular embodimentsdescribed herein. Additionally, either scheme or both schemes can becombined with the method employed above utilizing analysis of patches.

An example-based method can be utilized alternatively or additionally todefine a clear path based upon an input image. An exemplaryexample-based method collects a number of sample images of views,defining a clear path for each of the sample images, matches a currentimage to one or more of the sample images, and determines a clear pathbased upon the matching. Matching the current image to one or more ofthe sample images can be accomplished, for example, by extractingfeatures from each of the sample images, extracting features from thecurrent image, comparing the extracted features from the current imageto a database of extracted features from the sample images, andselecting matching sample images to the current image. A clear path canbe selected from the best matching sample image or can be determinedbased upon a combination of the closest matches to the current image.

Hierarchical configurations of multiple clear path detection algorithmsmay be incorporated into this disclosure, including algorithms arrangedbased upon computational intensity. Less computationally intensivedetection methods can identify clear paths in the image and leaveremaining sections of the image not identified as a clear path for morecomputationally intensive analysis thereby increasing computationalefficiency for clear path identification. The particular methodsutilized and the hierarchical structure of the particular hierarchicalmethod used can differ or change. In an exemplary hierarchicalconfiguration, a patch-based clear path detection method identifiesclear paths in the image before a pixel-based clear path detectionmethod analyzes the remaining sections of the image not identified as aclear path by the patch-based method. In one embodiment, a secondexemplary hierarchical layer utilizing an example-based method furtheranalyzes sections not identified as a clear path by the pixel-basedmethod. However, a number of different hierarchical configurations areenvisioned, and the disclosure is not intended to be limited to theparticular embodiments described herein.

The above described textureless method of image analysis filters from animage non-conforming regions of the image as not belonging to a planar,consistent road surface. This method, using the various filters appliedin parallel to the image, removes from the image areas that cannot be aclear path, and what remains is a candidate for designation as a clearpath. An alternative method, a segmentation-based clear path method,similarly analyzes with several analysis methods in parallel the imageto identify a clear path. However, instead of filtering away portions ofthe image, the exemplary segmentation-based method seeks to firstsubdivide or segment the image according to discernable boundarieswithin the image and to second make judgments regarding the segments toidentify a clear path.

A segmentation-based method uses analysis methods to subdivide an image.One exemplary configuration utilizes motion analysis, texture analysis,color analysis, and geometric analysis to segment the image.

Motion analysis can take many forms. As depicted in FIGS. 6A-6C,differencing between two images or a series of images can be utilized todistinguish movement of an object with respect to a background. Inanother example, feature recognition within the image can be utilized toevaluate a visible object to conform with a shape known to be prone tomovement, such as another vehicle oriented in a particular relativedirection to the vehicle, or a pedestrian. In another example, whereinanother vehicle system in addition to the image can be utilized, such asa radar system, providing radar return data from the object, or vehicleto vehicle (V2V) communications, providing position and movement datafrom the communicating vehicle, tracking and movement of the object canbe discerned and overlaid with the image to impose a subdivision uponthe image. One example of this would include opposing traffic upon atwo-way street. Sensing the movement of that traffic can be used tosegment the opposing lane from a current lane. Other forms of motionanalysis are known in the art. It will be appreciated that motion of thevehicle can be taken into account when judging motion of objects orfeatures within the image.

Once motion analysis is performed, objects or regions of the imageassociated with the motion relative to a stationary ground plane can besegmented or subdivided from the rest of the image as a region of theimage not likely to be a candidate for a clear path. Such exemplaryoperation segments a portion of the image as a stationary areapotentially containing a clear path from the region with identifiedmotion. Further, an implication of the detected motion can be utilized,for example, to segment a portion in front of the moving object as apotential collision zone and, therefore, not a clear path from otherportions of the image that can represent a clear path. A number ofmethods to employ information discernable through motion analysis areenvisioned, and the disclosure is not intended to be limited to theparticular embodiments described herein.

Texture analysis is discussed in association with the texture rich andtextureless methods described above. According to exemplary texture richmethods, pixel intensity contrasts, color contrasts, recognizable lines,corners and other features can all be recognized and analyzed in animage according to methods described above and according to othermethods known in the art. According to exemplary textureless methods,different filters can be applied to the image based upon recognizablepatterns in the image to identify areas in the image more likely toinclude a clear path.

Once texture analysis is performed, analysis of features apparent and/ortextureless regions in the pixels of the image can provide definition ofportions of the image useful to segment the image. Such exemplaryoperation segments a portion of the image based upon properties detectedand potential impacts to potential clear paths. Presence of particulartextures or pixelated features can be useful to analysis. For example,lane markers can be discerned and are useful to define differentsegments or sub-divisions of the image to represent the road surface andrelevant to defining a clear path. Similarly, curbsides, road shoulders,and roadside barriers can be used to segment a portion of the imagerepresenting the road surface from other areas of the image. Similarly,as described above, lane geometry or other indications that can bedetermined through texture analysis can be useful to define a horizon orvanishing point. The horizon or vanishing point can be used to segmentthe ground upon which a clear path may exist from sky and otherbackground above the horizon upon which a clear path may not exist.Additionally, objects discernable through texture rich analysis can beanalyzed according to their height relative to the surface of theground. Through this analysis, texture describing a median of the road,a snow bank, or a line of parked cars can be used to segment a region ofthe image upon which a clear path cannot exist from a region upon whicha clear path can exist. In another example, a lack of texture oridentification of a textureless region of the image, as a flat surfaceof a roadway can appear, can be useful to identify a segmented portionof the image as a potential clear path from other areas with discernabletexture. A number of methods to employ information discernable throughtexture analysis are envisioned, and the disclosure is not intended tobe limited to the particular embodiments described herein.

Color analysis can be employed, in methodology similar to thetextureless methods described above, to segment a portion of the imagethat can represent a road surface upon which a clear path may exist fromareas that cannot represent a road surface. Whereas the texturelessmethod filters or eliminates portions of an image based upon color,color analysis segments portions of the image based upon color,specifically segmenting portions of the image with colors that canrepresent a road surface from portions of the image with colors thatcannot represent a road surface.

Once color analysis is performed, regions of the image with colors thatcan represent a road surface can be distinguished from areas of the roadthat cannot represent a road surface. Color analysis can segmentportions of the image by color, for example, segmenting a green area ofthe image from a gray area of the image. In this example, a road can begray, whereas a road is unlikely to be green. Color analysis cansimilarly be used to define lane markers, construction zone markers,school zone markers, hatched designs upon the road indicating do nottravel zones and other indications that can be judged according to colorof markings upon or near the road. A number of methods to employinformation discernable through color analysis are envisioned, and thedisclosure is not intended to be limited to the particular embodimentsdescribed herein.

Segmentation of the image into a variety of subdivisions by theexemplary methods above and/or recognition of shapes evident within theimage of allow recognition of geometric patterns significant within theimage. Such geometric patterns can, once identified, be analyzed forsignificance to existence of a clear path.

Geometric shapes can be utilized to identify regions of the image likelyto indicate a road surface capable of being a clear path. For instance,a parallelogram-based shape wider at the base and narrower at the top,with substantially parallel lines including the base of the image and adetermined horizon line can be indicative of a current lane of travelupon a roadway. A similar parallelogram-based shape or a shape adjacentto a current lane of travel that could be a parallelogram if extendedpast an intervening vertical edge of the image or shapes with linesseemingly parallel to a current lane of travel can be indicative of aneighboring lane of travel and potentially a clear path depending uponother indications. Shapes contiguous with a current lane of travel or anidentified neighboring lane of travel, not segmented from the respectivelane of travel, can potentially be a connecting roadway and potentialclear path. Further, shapes can be logically joined together to indicatea road surface or lane of travel. For example, a transition from aroadway to a bridge surface frequently includes a noticeable transition.Such a transition, identified through the methods described above, cancreate geometric shapes terminating at the transition. However, analysisof the geometric shapes can indicate that the two shapes together likelyindicate a contiguous lane of travel.

Similarly, geometric shapes can be identified as not being indicative ofa roadway or a lane of travel capable of being a clear path. Lanes oftravel require that a vehicle can travel through the lane. Shapes endingabruptly, clearly separate from other identified lanes of travel,separated by a shape indicating an abrupt change in surface height orotherwise indicating obstruction to travel can be used to segmentportions of an image from other areas that can be a clear path.Exemplary shapes indicating abrupt changes in height can include shapesexhibiting vertical lines consistent with walls or sharp curbs.

Additionally, a size of an identified geometric shape can be descriptiveof whether the shape can represent a roadway or clear path. A lane oftravel close to the vehicle must be at least a certain size in order tobe a clear path through which the vehicle can travel. A geometric shapeor a pattern of geometric shapes can be determined to support or notsupport a clear path due to the potential size of the road surfaceindicated by the shapes. As the analyzed surface is further away fromthe vehicle, a lane of travel supporting a clear path can appear smallerin the image due to perspective. A number of methods to employinformation discernable through shape analysis are envisioned, and thedisclosure is not intended to be limited to the particular embodimentsdescribed herein.

It will be appreciated that the methods described above are exemplary.Fewer analyses can be combined to describe the clear path, and theparticular analyses selected can be selectably utilized based uponmonitored factors affecting the accuracy or efficiency of each of themethods according to any selection method sufficient to evaluate thedifferent analysis methods. Such factors can include weather, lightlevel, traffic density, speed of the vehicle, and other factors.

Different methods can additionally be utilized. For example, simplethresholding methods are envisioned. One example utilizes histogramthresholding segmentation. Such an exemplary method finds histogramvalley/valleys location(s) and uses the location(s) as threshold topartition the image. In another example, auto-thresholding segmentationor ostu-segmentation methods can be utilized. Instead of using apredefined fixed threshold, ostu-segmentation utilize methods sufficientto contemplate image recognition or partition methodology to determineoptimal thresholds to segment the image. Such threshold(s) can be foundusing a method to find an optimal threshold based on inter-class andinner-class variances.

Another example utilizes region growth segmentation. Beginning from aseed and spreading over the whole image, when a neighbor pixel isinvestigated, the difference is calculated from a previously determinedregion mean. If the difference is less than a threshold, then the pixelis merged into the region. Otherwise, this pixel forms a new seed andstarts another region.

Another example utilizes watershed methods to segment an image. In amethod analogous to following water flow on a topographic map, startpartitions are defined from the local lowest locations and thepartitions are increased until the each partitioned area reaches thewatershed ridges and stops there. These watershed ridges become theboundaries of the segmentation patches. Such a watershed method can workon original image or gradient image.

Another example includes clustering methods. For example, a K-meanmethod is known. In feature space, start from a randomly selected k orpre-defined center points. Clustering is then performed for all pixels.Through a series of iterations, a mean is updated (with a center infeature space) and a variance is updated, and clustering is againperformed in next iteration. Repetition stops when inner class variancereaches a threshold or after a maximal number of iterations of innerclass variance reaches a threshold and after clustering results do notchange from one iteration to the next. In another example, an ISODATAsegmentation method is known. ISODATA segmentation is an improvementover k-mean methods: instead of a fixed k partitions, ISODATA methodintroduces “split and merge” methods known in the art into theclustering process. When an inter-class distance is smaller than athreshold, then the classes are merged. When inner-class variance islarger than a threshold, then the class splits. The final partition canhave various patch numbers.

Another example includes graph based segmentation. The method can beperformed on grid-graph or image space or on a nearest-neighbor graph infeature space. When the method is performed in feature space, the resultis similar to K-mean or ISODATA methods. When the method is performed ongrid-graph, the method considers an image as a graph G=(V,E), where eachnode v_i in V corresponds to a pixel, and E contains edges connectingpairs of “neighboring” (in image space or in feature space) pixels. Theprinciple of this method is to compare the internal difference of acomponent and difference between two components, with the adjustment ofa tolerance function. The method begins from the smallest components(for example, starting with each pixel) and merges components until theinter-component difference is larger than the intra-componentdifference. The tolerance function controls the desired segmentationsize.

In another example, radar, LIDAR, global positioning data in combinationwith a digital map, vehicle to vehicle communication, vehicle toinfrastructure communication, or other inputs describing an operatingenvironment of the vehicle can be used to provide distinct analysis thatcan be overlaid upon and used to segment the input image. Other methodsof image processing or analysis or other sources of informationregarding the operating environment of the vehicle can be employed inaddition or in the alternative to those given as examples herein tosegment portions of an image that can support a clear path from otherportions that cannot, and the disclosure is not intended to be limitedto the exemplary embodiments described herein.

Additional methods are herein disclosed for detecting a clear path oftravel for the vehicle 100 including methods to detect objects aroundthe vehicle and place those objects into a context of a clear pathdetermination. Detection of the object or objects and any contextualinformation that can be determined can be utilized to enhance the clearpath of travel with an impact of object to the clear path. Informationregarding an operational environment of the vehicle, including detectionof objects around the vehicle, can be used to define or enhance a clearpath. For example, the clear path detection analysis may be augmentedusing vehicle detection analysis and construction area detectionanalysis. A detected object such as another vehicle or a constructionbarrier may inhibit safe vehicle travel and/or indicate conditions orconstrains limiting vehicle travel. The vehicle detection analysisdetects vehicles in a field-of-view. The construction area detectionanalysis detects construction areas and other zones associated withlimiting vehicle travel. The separate methods for analyzing a roadwayare used to strengthen confidence in clear paths identified in thefield-of-view. Similarly, clear path detection analysis can be used toaugment the vehicle detection analysis and the construction areadetection analysis.

Vehicle detection analysis includes analyzing the environment around thevehicle to detect other vehicles using one or multiple detectionsystems. The detection systems can include but are not limited tocamera-based systems, radar-based systems, LIDAR-based systems, andultrasonic sensor-based systems. It will be further appreciated thatdetection or relative location information can be remotely communicatedthrough V2V communications or vehicle to infrastructure (V2I)communications. Vehicle detection can include vehicle tracks todetermine vehicles as objects to be avoided. Additionally, vehicletracks can be used to describe exemplary clear paths and compare toother available information. Additionally, vehicle position with respectto the vehicle, for example, as described with respect to a longitudinalaxis of the host vehicle, can describe potential behavior of the clearpath based upon a single image. For example, a vehicle at some distancefrom the host vehicle, traveling away from the host vehicle, and located5 degrees to the left of the axis of the host vehicle, can be used toestimate curvature to the left in the upcoming roadway in anticipationof defining the clear path.

Methods are herein disclosed for detecting vehicles in an image using acamera-based vehicle detection system, described with reference to FIG.12. FIG. 12 shows an exemplary image 500. The camera-based vehicledetection system analyzes the image 500 in a number of ways to identifyimage coordinates corresponding to a vehicle. Texture-rich methods maybe used wherein corresponding matched interest points of sequentialimages are analyzed using triangulation methods to determine motion ofobjects associated with the corresponding matched interest points. Themotion may be evaluated for identifiable behavioral characteristics todetermine whether the object corresponds to a vehicle. The identifiablebehavioral characteristics correspond to behaviors attributable to avehicle, and may be assigned a confidence likelihood value, wherein anobject corresponding to a confidence likelihood exceeding a threshold isdefined as a vehicle. The identifiable behavioral characteristics may beevaluated in context of image location and speed of the vehicle 100. Forexample, objects corresponding to a region immediately in front of thevehicle will likely have a speed similar to the vehicle 100 as comparedwith subsequently monitored images. Additional detected stationary ortransient objects may aid the evaluation including detected laneboundaries 505 and/or lane markers 506. Objects that correspond to avehicle are more likely to travel within the lane marker and avoidcrossing lane boundaries. Objects exhibiting identifiable behaviorsincrease a confidence likelihood that the objects correspond to avehicle or vehicles.

Objects exhibiting motion and identified using texture-rich methods maybe used in conjunction with template matching programming. Templatematching includes comparing predetermined templates with an image area504 selected in the image. Image areas within the image include aplurality of spatially related pixels selected for analysis andidentifiable by image coordinates. Template matching can be executedusing one of multiple methods, including on gray-scale images and lightnormalized images. A plurality of vehicle templates is constructed tocorrespond to multiple views of a vehicle, including templatesassociated with longitudinal vehicle travel and templates associatedwith substantially perpendicular crossing vehicle travel. The templatesare compared with the image area corresponding to the feature exhibitingmotion. In one embodiment, template matching may include comparing imageintensities of pixels with the template using a sum of absolutedifferences measurement. Linear spatial filtering may be executed on theimage area to determine an image area with a highest correspondence withthe template. Image areas associated with a predetermined thresholdcorrespondence are considered to match the template.

Detecting objects around a vehicle frequently include vehicle tracking.An exemplary method to accomplish vehicle tracking is disclosed. Aradar-based vehicle detection system uses the radar imaging system 130to estimate vehicle dimensions and motion of vehicles in front of thevehicle 100. Returns can be analyzed and vehicle locations with respectto the vehicle 100 can be estimated according to methods known in theart.

Referring to FIGS. 15-17, the vehicle 100 includes radar imaging system130 including a target tracking system including sensing devices 714 and716 to estimate vehicle dimensions and motion of vehicles in front ofthe vehicle 100. FIG. 15 shows a schematic diagram of the vehicle 100system which has been constructed with the target tracking system, inaccordance with the present disclosure. The exemplary target trackingsystem preferably includes object-locating sensors comprising at leasttwo forward-looking range sensing devices 714 and 716. Theobject-locating sensors may include a short-range radar subsystem, along-range radar subsystem, and a forward vision subsystem. Theobject-locating sensing devices may include any range sensors, such asFM-CW radars, (Frequency Modulated Continuous Wave), pulse and FSK(Frequency Shift Keying) radars, and LIDAR (Light Detection and Ranging)devices, and ultrasonic devices which rely upon effects such asDoppler-effect measurements to locate forward objects.

These sensors are preferably positioned within the vehicle 100 inrelatively unobstructed positions relative to a view in front of thevehicle. It is also appreciated that each of these sensors provides anestimate of actual location or condition of a targeted object, whereinthe estimate includes an estimated position and standard deviation. Assuch, sensory detection and measurement of object locations andconditions are typically referred to as “estimates.” It is furtherappreciated that the characteristics of these sensors are complementary,in that some are more reliable in estimating certain parameters thanothers. Conventional sensors have different operating ranges and angularcoverages, and are capable of estimating different parameters withintheir operating range. For example, radar sensors can usually estimaterange, range rate and azimuth location of an object, but is not normallyrobust in estimating the extent of a detected object. A camera withvision processor is more robust in estimating a shape and azimuthposition of the object, but is less efficient at estimating the rangeand range rate of the object. Scanning type LIDARs perform efficientlyand accurately with respect to estimating range, and azimuth position,but typically cannot estimate range rate, and is therefore not accuratewith respect to new object acquisition/recognition. Ultrasonic sensorsare capable of estimating range but are generally incapable ofestimating or computing range rate and azimuth position. Further, it isappreciated that the performance of each sensor technology is affectedby differing environmental conditions. Thus, conventional sensorspresent parametric variances, but more importantly, the operativeoverlap of these sensors creates opportunities for sensory fusion.

Each object-locating sensor and subsystem provides an output includingrange, R, time-based change in range, R_dot, and angle, Θ, preferablywith respect to a longitudinal axis of the vehicle, which can be writtenas a measurement vector (o), i.e., sensor data. An exemplary short-rangeradar subsystem has a field-of-view (FOV) of 160 degrees and a maximumrange of thirty meters. An exemplary long-range radar subsystem has afield-of-view of 17 degrees and a maximum range of 220 meters. Anexemplary forward vision subsystem has a field-of-view of 45 degrees anda maximum range of fifty (50) meters. For each subsystem thefield-of-view is preferably oriented around the longitudinal axis of thevehicle 100. The vehicle 100 is preferably oriented to a coordinatesystem, referred to as an XY-coordinate system 720, wherein thelongitudinal axis of the vehicle 100 establishes the X-axis, with alocus at a point convenient to the vehicle and to signal processing, andthe Y-axis is established by an axis orthogonal to the longitudinal axisof the vehicle 100 and in a horizontal plane, which is thus parallel toground surface.

FIG. 16 shows a control system for information flow utilized in creatinga track list, in accordance with the present disclosure. The controlsystem includes an observation module 722, a data association andclustering (DAC) module 724 that further includes a Kalman filter 724A,and a track life management (TLM) module 726 that keeps track of a tracklist 726A comprising of a plurality of object tracks. More particularly,the observation module 722 operates sensors 714 and 716, theirrespective sensor processors, and the interconnection between thesensors, sensor processors, and the DAC module 724.

FIG. 17 depicts an exemplary data fusion process, in accordance with thepresent disclosure. As shown in FIG. 17, the illustrated observationmodule includes first sensor 714 located and oriented at a discretepoint A on the vehicle, first signal processor 714A, second sensor 716located and oriented at a discrete point B on the vehicle, and secondsignal processor 716A. The first processor 714A converts signals(denoted as measurement o_(A)) received from the first sensor 714 todetermine range (RA), a time-rate of change of range (R_dotA), andazimuth angle (ΘA) estimated for each measurement in time of targetobject 730. Similarly, the second processor 716A converts signals(denoted as measurement o_(B)) received from the second sensor 716 todetermine a second set of range (RB), range rate (R_dotB), and azimuthangle (ΘB) estimates for the object 730.

The exemplary DAC module 724 includes a controller 728, wherein analgorithm and associated calibration is stored and configured to receivethe estimate data from each of the sensors 714 and 716 to cluster datainto like observation tracks (i.e. time-coincident observations of theobject 730 by sensors 714 and 716 over a series of discrete timeevents), and to fuse the clustered observations to determine a truetrack status. It is understood that fusing data using different sensingsystems and technologies yields robust results. Again, it is appreciatedthat any number of sensors can be used in this technique. However, it isalso appreciated that an increased number of sensors results inincreased algorithm complexity, and the requirement of more computingpower to produce results within the same time frame. The controller 728is housed within the host vehicle 100, but may also be located at aremote location. In this regard, the controller 728 is electricallycoupled to the sensor processors 714A, 716A, but may also be wirelesslycoupled through RF, LAN, infrared or other conventional wirelesstechnology. The TLM module 726 is configured to receive and store fusedobservations in a list of tracks 726A.

Sensor registration, or “alignment” of sensors, in multi-target tracking(‘MTT’) fusion, involves determining the location, orientation andsystem bias of sensors along with target state variables. In a generalMTT system with sensor registration, a target track is generated duringvehicle operation. A track represents a physical object and comprises anumber of system state variables, including, e.g., position andvelocity. Measurements from each individual sensor are usuallyassociated with a certain target track. A number of sensor registrationtechniques are known in the art and will not be discussed in detailherein.

The schematic illustration of FIG. 15 includes the aforementionedobject-locating sensors 714 and 716 mounted on the exemplary vehicle atpositions A and B, preferably mounted at the front of the vehicle 100.The target object 730 moves away from the vehicle, wherein t1, t2, andt3 denote three consecutive time frames. Lines ra1-ra2-ra3, rf1-rf2-rf3,and rb1-rb2-rb3 represent, respectively, the locations of the targetmeasured by first sensor 714, fusion processor, and second sensor 716 attimes t1, t2, and t3, measured in terms of o_(A)=(R_(A), R_dot_(A),Θ_(A)) and o_(B)=(R_(B), R_dot_(B), Θ_(B)), using sensors 714 and 716,located at points A, B.

A known exemplary trajectory fusing process permits determining positionof a device in the XY-coordinate system relative to the vehicle. Thefusion process comprises measuring the target object 730 in terms ofo_(A)=(R_(A), R_dot_(A), Θ_(A)) and o_(B)=(R_(B), R_dot, Θ_(B)), usingsensors 714 and 716, located at points A, B. A fused location for thetarget object 730 is determined, represented as x=(RF, R_dotF, ΘF,ΘF_dot), described in terms of range, R, and angle, Θ, as previouslydescribed. The position of forward object 730 is then converted toparametric coordinates relative to the vehicle's XY-coordinate system.The control system preferably uses fused track trajectories (Line rf1,rf2, rf3), comprising a plurality of fused objects, as a benchmark,i.e., ground truth, to estimate true sensor positions for sensors 714and 716. As shown in FIG. 15, the fused track's trajectory is given bythe target object 730 at time series t1, t2, and t3. Using a largenumber of associated object correspondences, such as {(ra1, rf1, rb1),(ra2, rf2, rb2), (ra3, rf3, rb3)} true positions of sensors 714 and 716at points A and B, respectively, can be computed to minimize residues,preferably employing a known least-squares calculation method. In FIG.15, the items designated as ra1, ra2, and ra3 denote an object mapmeasured by the first sensor 714. The items designated as rb1, rb2, andrb3 denote an object map observed by the second sensor 716.

In FIG. 16, referenced tracks are preferably calculated and determinedin the sensor fusion block 728 of FIG. 17, described above. The processof sensor registration comprises determining relative locations of thesensors 714 and 716 and the relationship between their coordinatesystems and the frame of the vehicle, identified by the XY-coordinatesystem. Registration for single object sensor 716 is now described. Allobject sensors are preferably handled similarly. For object mapcompensation, the sensor coordinate system or frame, i.e. theUV-coordinate system, and the vehicle coordinate frame, i.e. theXY-coordinate system, are preferably used. The sensor coordinate system(u, v) is preferably defined by: (1) an origin at the center of thesensor; (2) the v-axis along longitudinal direction (bore-sight); and(3) a u-axis normal to v-axis and points to the right. The vehiclecoordinate system, as previously described, is denoted as (x, y) whereinx-axis denotes a vehicle longitudinal axis and y-axis denotes thevehicle lateral axis.

The locations of track (x) can be expressed in XY-coordinate system as(r). Sensor measurement (o) can be expressed in UV-coordinate as (q).The sensor registration parameters (a) comprise of rotation (R) andtranslation (r0) of the UV-coordinate system.

FIG. 18 depicts an exemplary dataflow enabling joint tracking and sensorregistration, in accordance with the present disclosure. The method isinitiated upon reception of sensor data. A data association module willmatch the sensor data with the predicted location of a target. The jointtracking and registration module combines the previous estimation (i.e.,a priori) and new data (i.e., matched measurement-track pairs), andupdates the target tracks estimation and sensor registration data in thedatabase. The time propagation process module predicts the target tracksor sensor registration parameters in the next time cycle based on thehistorical sensor registration, tracks and current vehicle kinematicsvia a dynamics model. The sensor registration parameters are usuallyassumed to be substantially constant over time. Confidence of theregistration parameters accumulates over time. However, a prioriinformation about registration will be reset to zero when a significantsensor registration change is detected (e.g., vehicle collision).

Object tracks can be utilized for a variety of purposes includingadaptive cruise control, wherein the vehicle adjusts speed to maintain aminimum distance from vehicles in the current path, as described above.Another similar system wherein object tracks can be utilized is acollision preparation system (CPS), wherein identified object tracks areanalyzed in order to identify a likely impending or imminent collisionbased upon the track motion relative to the vehicle. A CPS warns thedriver of an impending collision and reduces collision severity byautomatic braking if a collision is considered to be unavoidable. Amethod is disclosed for utilizing a multi-object fusion module with aCPS, providing countermeasures, such as seat belt tightening, throttleidling, automatic braking, air bag preparation, adjustment to headrestraints, horn and headlight activation, adjustment to pedals or thesteering column, adjustments based upon an estimated relative speed ofimpact, adjustments to suspension control, and adjustments to stabilitycontrol systems, when a collision is determined to be imminent.

FIG. 19 schematically illustrates an exemplary system whereby sensorinputs are fused into object tracks useful in a collision preparationsystem, in accordance with the present disclosure. Inputs related toobjects in an environment around the vehicle are monitored by a datafusion module. The data fusion module analyzes, filters, or prioritizesthe inputs relative to the reliability of the various inputs, and theprioritized or weighted inputs are summed to create track estimates forobjects in front of the vehicle. These object tracks are then input tothe collision threat assessment module, wherein each track is assessedfor a likelihood for collision. This likelihood for collision can beevaluated, for example, against a threshold likelihood for collision,and if a collision is determined to be likely, collisioncounter-measures can be initiated.

As shown in FIG. 19, a CPS continuously monitors the surroundingenvironment using its range sensors (e.g., radar, LIDAR, ultrasonicsensors) and cameras and takes appropriate counter-measurements in orderto avoid incidents or situations to develop into a collision. Acollision threat assessment generates output for the system actuator torespond.

As described in FIG. 19, a fusion module is useful to integrate inputfrom various sensing devices and generate a fused track of an object infront of the vehicle. The fused track created in FIG. 19 comprises adata estimate of relative location and trajectory of an object relativeto the vehicle. This data estimate, based upon radar and other rangefinding sensor inputs is useful, but includes the inaccuracies andimprecision of the sensor devices utilized to create the track. Asdescribed above, different sensor inputs can be utilized in unison toimprove accuracy of the estimates involved in the generated track. Inparticular, an application with invasive consequences such as automaticbraking and potential airbag deployment require high accuracy inpredicting an imminent collision, as false positives can have a highimpact of vehicle drivability, and missed indications can result ininoperative safety systems.

Vision systems provide an alternate source of sensor input for use invehicle control systems. Methods for analyzing visual information areknown in the art to include pattern recognition, corner detection,vertical edge detection, vertical object recognition, and other methods.However, it will be appreciated that high-resolution visualrepresentations of the field in front a vehicle refreshing at a highrate necessary to appreciate motion in real-time include a very largeamount of information to be analyzed. Real-time analysis of visualinformation can be prohibitive. A method is disclosed to fuse input froma vision system with a fused track created by methods such as theexemplary track fusion method described above to focus vision analysisupon a portion of the visual information most likely to pose a collisionthreat and utilized the focused analysis to alert to a likely imminentcollision event.

FIG. 20 schematically illustrates an exemplary image fusion module, inaccordance with the present disclosure. The fusion module of FIG. 20monitors as inputs range sensor data comprising object tracks and cameradata. The object track information is used to extract an image patch ora defined area of interest in the visual data corresponding to objecttrack information. Next, areas in the image patch are analyzed andfeatures or patterns in the data indicative of an object in the patchare extracted. The extracted features are then classified according toany number of classifiers. An exemplary classification can includeclassification as a fast moving object, such a vehicle in motion, a slowmoving object, such as a pedestrian, and a stationary object, such as astreet sign. Data including the classification is then analyzedaccording to data association in order to form a vision fused basedtrack. These tracks and associated data regarding the patch are thenstored for iterative comparison to new data and for prediction ofrelative motion to the vehicle suggesting a likely or imminent collisionevent. Additionally, a region or regions of interest, reflectingpreviously selected image patches, can be forwarded to the moduleperforming image patch extraction, in order to provide continuity in theanalysis of iterative vision data. In this way, range data or rangetrack information is overlaid onto the image plane to improve collisionevent prediction or likelihood analysis.

FIG. 22 illustrates exemplary range data overlaid onto a correspondingimage plane, useful in system-internal analyses of various targetobjects, in accordance with the present disclosure. The shaded bars arethe radar tracks overlaid in the image of a forward-looking camera. Theposition and image extraction module extracts the image patchesenclosing the range sensor tracks. The feature extraction modulecomputes the features of the image patches using following transforms:edge, histogram of gradient orientation (HOG), scale-invariant featuretransform (SIFT), Harris corner detectors, or the patches projected ontoa linear subspace. The classification module takes the extractedfeatures as input and feeds to a classifier to determine whether animage patch encloses an object. The classification determines the labelof each image patch. For example, in FIG. 22, the boxes A and B areidentified as vehicles while the unlabelled box is identified as aroad-side object. The prediction process module utilizes an object'shistorical information (i.e., position, image patch, and label ofprevious cycle) and predicts the current values. The data associationlinks the current measurements with the predicted objects, or determinesthe source of a measurement (i.e., position, image patch, and label) isfrom a specific object. In the end, the object tracker is activated togenerate updated position and save back to the object track files.

FIG. 21 schematically depicts an exemplary bank of Kalman filtersoperating to estimate position and velocity of a group objects,accordance with the present disclosure. Different filters are used fordifferent constant coasting targets, high longitudinal maneuver targets,and stationary targets. A Markov decision process (MDP) model is used toselect the filter with the most likelihood measurement based on theobservation and target's previous speed profile. This Multi-modelfiltering scheme reduces the tracking latency, which is important forCPS function.

Reaction to likely collision events can be scaled based upon increasedlikelihood. For example, gentle automatic braking can be used in theevent of a low threshold likelihood being determined, and more drasticmeasures can be taken in response to a high threshold likelihood beingdetermined.

Additionally, it will be noted that improved accuracy of judginglikelihood can be achieved through iterative training of the alertmodels. For example, if an alert is issued, a review option can be givento the driver, through a voice prompt, and on-screen inquiry, or anyother input method, requesting that the driver confirm whether theimminent collision alert was appropriate. A number of methods are knownin the art to adapt to correct alerts, false alerts, or missed alerts.For example, machine learning algorithms are known in the art and can beused to adaptively utilize programming, assigning weights and emphasisto alternative calculations depending upon the nature of feedback.Additionally, fuzzy logic can be utilized to condition inputs to asystem according to scalable factors based upon feedback. In this way,accuracy of the system can be improved over time and based upon theparticular driving habits of an operator.

It will be appreciated that similar methods employed by the CPS can beused in a collision avoidance system. Frequently such systems includewarnings to the operator, automatic brake activation, automatic lateralvehicle control, changes to a suspension control system, or otheractions meant to assist the vehicle in avoiding a perceived potentialcollision.

Additionally, numerous methods are known to achieve lane keeping orplace a vehicle within a lane by sensor inputs. For example, a methodcan analyze visual information including paint lines on a road surface,and utilize those markings to place the vehicle within a lane. Somemethods utilize tracks of other vehicles to synthesize or assist inestablishing lane geometry in relation to the vehicle. GPS devices,utilized in conjunction with 3D map databases, make possible estimatinga location of a vehicle according to global GPS coordinates andoverlaying that position with known road geometries.

An exemplary method for generating estimates of geometry of a lane oftravel for a vehicle on a road is disclosed. The method includesmonitoring data from a global positioning device, monitoring mapwaypoint data describing a projected route of travel based upon astarting point and a destination, monitoring camera data from a visionsubsystem, monitoring vehicle kinematics data including: a vehiclespeed, and a vehicle yaw rate, determining a lane geometry in an area ofthe vehicle based upon the map waypoint data and a map database,determining a vehicle position in relation to the lane geometry basedupon the lane geometry, the data from the global positioning device, andthe camera data, determining a road curvature at the vehicle positionbased upon the vehicle position, the camera data, and the vehiclekinematics data, determining the vehicle orientation and vehicle lateraloffset from a center of the lane of travel based upon the roadcurvature, the camera data and the vehicle kinematics, and utilizing thevehicle position, the road curvature, the vehicle orientation, and thevehicle lateral offset in a control scheme of the vehicle.

A LIDAR-based vehicle detection system can additionally or alternativelybe utilized, transmitting and receiving optical energy to estimatevehicle dimensions and motion of vehicles in front of the vehicle 100.Similar to the radar-based vehicle detection system describedhereinabove, patterns in the light waves reflecting off these vehiclescan be analyzed and vehicle locations with respect to the vehicle 100can be estimated.

Vehicle detection analysis is preferably augmented using lane geometrydetection analysis to identify lane markers and/or road edges in afield-of-view. The lane markers 506 may be used to define vehicle laneboundaries and suggest image coordinates corresponding to greaterlikelihood for correspondence detected vehicles on a roadway. The lanemarkers 506 may be used to predict vehicle travel on the roadway,thereby assisting the clear path detection analysis. The lane markerdetection analysis may be repeated or reiterated in a number of ways,with the same image area being analyzed repeatedly with the selectionand analysis of different image areas, and an identified feature trackedand analyzed for change over a number of sequential images.

Lane markers may be detected using one of multiple methods. Camera-baseddetection includes convolving image data using a plurality of lineorientation filters to generate a filter response that identifies anorientation of each segment of a lane marker. In one embodiment, a Houghtransform technique is applied to identify line segments of lane markersusing points on the lane markers. The Hough transform techniqueevaluates the line segments to determine a confidence that the linesegment corresponds to a lane marker. Line segments corresponding to aconfidence greater than a threshold are defined as a lane marker. Thelane marker may be error tested based upon correspondence to identifiedobjects in the field-of-view including the ground 20, and error testedfor association with previously identified lane markers.

Another method to detect lane markers or road edges includes use of aLIDAR device. LIDAR utilizes an emission of light to generate sensorreadings based upon the return signal of the light. It will beappreciated that LIDAR can be used to detect ranges to detected objectsand in certain configurations can return three dimensional readingsdescribing the environment surrounding the vehicle. It will additionallybe appreciated that LIDAR can be utilized to analyze an intensity of thereturn light to describe the environment surrounding the vehicle. Sensorreadings from a LIDAR system can be utilized to define a road edgeaccording to a texture of three dimensional readings describing grassand other characteristics of an edge to the road. Also sensor readingsfrom a LIDAR system can be utilized to define lane markers, for example,as described by intensity differences evident between a road surface andreflective paint used for lane markers.

Construction area detection analysis includes detecting constructionareas and other zones associated with limiting vehicle travel. Travelmay be limited in construction zones by construction objects, e.g.,construction barrels, barriers, fencing and grids, mobile units,scattered debris, and workers. Travel may additionally be limited inconstruction zones by traveling constraints, e.g., speed zones andmerger requirements. Detection of a construction area may be useful forchanging vehicle control and operation in a number of ways. Upondetection of a construction area, vehicle programming may initiate orend certain autonomous or semi-autonomous vehicle controls. Vehicleoperation can be limited to posted vehicle speeds and travel limited tocertain areas or lanes. Detection of a construction zone may alsoinitiate certain programming algorithms or detection schemes including,for example, a construction worker detection algorithm useful in areaswhere construction worker presence necessitates a speed decrease for thevehicle 100.

Methods are herein disclosed for detecting construction area in animage, described with reference to FIG. 13. Construction area detectionanalysis can utilize object detection methods, similar to thosedescribed above for vehicle detection, including methods utilizing oneor more of a camera-based system, a radar-based system, and aLIDAR-based system. Construction area detection analysis can includeanalyzing an operational environment of the vehicle for correspondenceto predetermined templates. The templates are associated withconstruction objects in the field-of-view indicating the vehicle 100 islocated in a construction area.

FIG. 13 shows exemplary construction objects corresponding toconstruction barrels 604. The predetermined templates can include, forexample, construction barrels, construction cones, road-side concretegrooves, mobile unit warnings, lane shift instructions, lane changingprohibitions, and other similar construction area signs. Patternrecognition programming including character or pattern recognition andtraffic sign recognition can be utilized to comprehend the contextualinformation available from construction area signs. Such contextualinformation can be utilized, for example, to determine lane geometriesand restrictions for use in enhancing the clear path. During vehicleoperation, the processing module 120 can analyze image areas, forexample exemplary image area 602 depicted in FIG. 13, for correspondenceto the predetermined construction templates using template matchingprogramming, such as the type described hereinabove. The processingmodule 120 may scan for general outlines of a certain template beforeinitiating more computationally intense processing. For example, theprocessing module 120 may analyze the image for a sign silhouette beforeanalyzing the image for the type of sign. Identification of aconstruction template in the field-of-view indicates the vehicle is in aconstruction area.

Construction area detection analysis can further include analyzing animage 600 to determine image coordinates corresponding to a clear path.Construction templates as described hereinabove may be used to identifyconstruction area indicators corresponding to a clear path boundarybetween a clear path and a not clear path. Intermittent constructionobjects scattered on a roadway, including, for example, constructionbarrels 604 can indicate a clear path boundary 610. The processingmodule 120 identifies a line of trajectory indicated by the intermittentconstruction objects to identify the boundary. In one embodiment, pointscorresponding to a corner of a detected construction objected areconnected to determine the trajectory. The clear path boundary 610represents the demarcation between clear path and not clear path. Theconstruction area detection analysis identifies image areas enclosed bythe boundaries as corresponding to a clear path 615. In one embodiment,identification of the clear path boundary can be used to focus theanalysis on image area between identified boundaries to increasecomputational efficiency.

As described herein above, the clear path detection analysis may beaugmented with the vehicle detection analysis and the construction areadetection analysis. Multiple methods for augmenting the clear pathdetection analysis are contemplated by the disclosure, includingaugmenting the clear path detection analysis with both the vehicledetection analysis and the construction area detection analysis. Alikelihood analysis may be adopted to incorporate confidence valuesassigned for both the detected vehicles and features identified usingthe construction area detection analysis. However, the order of themethods used or disqualification or substitution of one of the methodscan be implemented based upon perceived factors affecting efficiency orperformance of the methods. For example, an urban environment withrapidly changing traffic conditions or pedestrian traffic may yieldparticularly poor results for one of the methods. An excess of roaddebris can be another example affecting one method more than others.Contamination of the image from the camera caused by a spot upon thelens of the camera may similarly cause poor results from one of themethods. Vehicle speed may affect the quality of results from one of themethods. Such factors can be determined experimentally, computationally,or be judged based upon modeling or any other method sufficient toaccurately estimate effectiveness of the various clear path methods todetermine a clear path based upon the input image. Additionally, methodssuch as machine learning algorithms or fuzzy logic can be implemented toadapt selection of the methods based upon in-vehicle use of the variousclear path methods and the resulting operation of the vehicle. Suchoperation of the vehicle can be judged in many ways known in the art,such as monitoring an occurrence of operator intervention in control ofthe vehicle or by monitoring location of lane markers in imagesdescribing subsequent control of the vehicle upon the roadway. A numberof methods to select among the methods and to adjust selection of thevarious clear path methods are envisioned, and the disclosure is notintended to be limited to particular exemplary embodiments disclosedherein.

Methods to detect objects around the vehicle and place those objectsinto a context of a clear path determination including vehicle detectionanalysis and construction area detection analysis are described above.It will be appreciated that similar methods can be employed determiningclear paths or enhancing clear paths based upon detection of otherobjects in the operating environment of the vehicle. For example, tollbooths on an expressway can be detected based upon signs warning of theupcoming booths and recognizable geometry of the booths and vehicle flowthrough the booths. In such a situation, a clear path can be selectablydefined leading to the rear of one of the lines of vehicles going thoughthe booths and limited to the area of the booth until a detectedturnpike or electronic signal from the booth indicates that the vehicleis free to proceed. In another example, a driver operating an autonomousor semi-autonomous vehicle could select an option to stop the vehiclefor refueling once the fuel tank has been depleted to a set level. Insuch a situation, a clear path can be defined based upon a perceivedavailable refueling station in the proximity of the vehicle andrecognizable geometry of the refueling station as it is approached,including the presence of other vehicles at different pumping stationswithin the refueling station. A number of exemplary uses of methods todetect objects around the vehicle and place those objects into a contextof a clear path determination are envisioned, and the disclosure is notintended to be limited to the particular exemplary embodiments disclosedherein.

FIG. 14 illustrates an exemplary method 800 for augmented clear pathdetection analysis, in accordance with the present disclosure. Themethod 800 is illustrated in FIG. 14, and described herein as comprisingdiscrete elements. Such illustration is for ease of description and itshould be recognized that the functions performed by these elements maybe combined in one or more devices, e.g., implemented in software,hardware, and/or application-specific integrated circuitry. For example,the method 800 may be executed as one or more algorithms in theprocessing module 120.

During operation, the camera 110 generates an image for analysis in theprocessing module 120 (802). The processing module 120 applies aparallel processing approach that includes concurrently analyzing theimage using the clear path detection analysis and one of the methods fordetection of objects around the vehicle described herein, for example,including vehicle detection analysis. The vehicle detection analysisdetects objects in the roadway (804). The clear path detection analysisidentifies image areas corresponding to a clear path (806). The outputsof steps 804 and 806 are compared and utilized to form an enhanced clearpath through mutual benefits refinement in step 808. Detection of theobjects in step 804 can simply include object positions or tracks forlater comparison to the clear path developed in step 806. Additionallyor alternatively, contextual analysis of the detected objects can beperformed in one of the illustrated steps, for example, allowingprojection of a future location of a detected vehicle or an otherwiseundetected lane geometry based upon behavior of detected vehicles.Detected objects can be used in the clear path detection analysis toenhance or refine image areas corresponding to a clear path. In oneexemplary embodiment, vehicle motion may be used to infer imagecoordinates corresponding to a clear path. Referring back to FIG. 12,exemplary inferred image coordinates 502 are inferred based on detectedmotion of a vehicle. Vehicle motion may be determined using one ofmultiple methods. An interest point corresponding to the vehicle may betracked over sequential images. Triangulation methods known in the artmay be used to determine vehicle trajectory based upon motion of thevehicle 100 and change in image coordinates of the interest point. LIDARand radar-based methods may also be used to determine a vehicles motion.Vehicle trajectory may be followed back to determine image areas whereinthe detected vehicle recently traversed. Image areas wherein a detectedvehicle has recently traversed are likely safe for vehicle travel andthus may appropriately be defined as corresponding to a clear path.Vehicle trajectories can be used to define or augment a clear path byfollowing the other vehicle, or vehicle trajectories, describing aprojected future location of the other vehicle, can be used to subtractfrom a clear path, avoiding potential collision with the other vehicle.

Step 808 of FIG. 14 illustrates mutual benefits refinement, inputting aclear path and object location or tracking information, to output anenhanced clear path. As will be appreciated from the above describedmethods, clear path can benefit from detection of objects around thevehicle, augmenting an understanding of what areas are likely to be aclear path. It will be appreciated that enhanced object location andtracking can similarly be accomplished by analysis of the identifiedclear path. For example, in a construction area, shapes detected atregular intervals along the border of the clear path can be assumed tobe construction barrels or other barriers, and methods to identify thebarrels can be focused upon the locations described by the clear pathdetection. Such an analysis can be iteratively reinforced withsubsequent images, building confidence along the route of travel of boththe clear path and the location and identification of the trackedobject. Additionally, a single image can be iteratively examined, withmultiple iterations of clear path analysis and object detectionreinforcing analysis of the single image. Such an analysis loop canbeneficially refine starting assumptions and results of the analyses.FIG. 23 graphically depicts such an iterative analysis, in accordancewith the present disclosure. Step 808 is depicted, including objecttracking refinement in step 805 and clear path refinement in step 807.Step 808 outputs both the enhanced clear path, as described above, andadditionally exemplary enhanced object tracking information.

In one embodiment of the clear path detection analysis, the detectedobjects may be utilized in the above described likelihood analysis. Theclear path confidence likelihood for each of the images areas arepartially based upon an association with the detected objects. Asdescribed hereinabove, the likelihood analysis evaluates image areas toassign a clear path confidence likelihood value. Classifiers may be usedto increase or decrease the clear path confidence likelihood value basedupon features identified in the image area. For example, detectedconstruction barriers or markers may be used as an additional classifierin accordance with the likelihood analysis set forth above to decreasethe clear path confidence likelihood of a corresponding image area.Detected vehicles tracked driving through detected construction barriersmay increase the clear path confidence likelihood of the image areacorresponding to where the vehicles were tracked. Image areas with aclear path confidence likelihood greater than a threshold confidencelikelihood are defined as clear path.

The clear path detection analysis may also be used to augment and refinean exemplary vehicle detection analysis. Image areas corresponding to aclear path define image coordinates devoid of detectable vehicles andthus can properly be excluded from the vehicle detection analysis,thereby improving computational efficiency. Similarly, a pocket of notclear path large enough to contain a vehicle within a larger imageregion of clear path is likely to include a vehicle, and computationalresources utilized in vehicle detection or tracking can be focused uponthat area.

Construction area indicators can be used in the clear path detectionanalysis to enhance or refine image areas corresponding to a clear pathaccording to additional exemplary embodiments. In one embodiment,construction area indicators are used to determine image coordinatescorresponding to a clear path. Construction area indicators areidentified by template matching programming. The processing module 120identifies a line of trajectory indicated by the intermittentconstruction objects to identify the clear path boundary. The clear pathboundaries are used to determine image coordinates corresponding to aclear path. Image coordinates enclosed by the boundaries are defined ascorresponding to a clear path. The image coordinates identified by theconstruction area detection analysis as corresponding to a clear pathcan be used by the clear path detection analysis to limit potentialimage coordinates corresponding to a clear path. In one embodiment, onlyimage coordinates identified by the construction area detection analysisare analyzed by the clear path detection analysis for inclusion on aclear path map.

The clear path detection analysis may also be used to augment and refinethe construction area detection analysis. In one embodiment, imagecoordinates identified by the clear path detection analysis ascorresponding to a clear path are excluded from the construction areadetection analysis.

It will be appreciated that a plurality of methods detecting objectsaround the vehicle can be utilized to simultaneously augment or enhancea clear path. According to one exemplary method, an enhanced clear pathwill include only areas upon a roadway indicated to be clear based uponall available methods. In another exemplary method, an enhanced clearpath will be decided according to a likelihood analysis for differentareas of an image, and each of the methods can be used to modify theresulting likelihood values for each area of the image. A number ofmethods to combine a plurality of methods detecting objects around thevehicle and a clear path analysis are envisioned, and the disclosure isnot intended to be limited to the particular exemplary embodimentsdescribed herein.

The above clear path analysis methods, including patch-based,pixel-based, example-based methods and segmentation-based, utilize acamera device to analyze an image or series of images. Additionally, themethods described to detect objects around the vehicle can includeanalysis of an image or a series of images. It will be appreciated thatthe image utilized in the clear path analysis can but need not be thesame image utilized in the method to detect objects around the vehicle.For example, different cameras might be utilized based upon theparticular requirements of the different analyses.

As mentioned above, processing module 120 may include algorithms andmechanisms to actuate autonomous driving control by means known in theart and not described herein, or processing module 120 may simplyprovide information to a separate autonomous driving system. Reactionsto perceived objects can vary, and include but are not limited tosteering changes, throttle changes, braking responses, and warning andrelinquishing control of the vehicle to the operator.

The disclosure has described certain preferred embodiments andmodifications thereto. Further modifications and alterations may occurto others upon reading and understanding the specification. Therefore,it is intended that the disclosure not be limited to the particularembodiment(s) disclosed as the best mode contemplated for carrying outthis disclosure, but that the disclosure will include all embodimentsfalling within the scope of the appended claims.

The invention claimed is:
 1. Method for detecting a clear path of travelfor a host vehicle including fusion of clear path detection by imageanalysis and detection of an object within an operating environment ofthe host vehicle, the method comprising: monitoring an image from acamera device; analyzing the image through clear path detection analysisto determine a clear path of travel within the image; monitoring twosets of sensor data describing the object, each set of sensor datacomprising a range, a time-rate of change of range, and an azimuth angleof the object; fusing, over a series of discrete time events, the twosets of sensor data to determine a tracked position and velocity of theobject; analyzing the tracked position and velocity of the object todetermine an impact of the object to the clear path, comprising:extracting a defined area of interest in the image corresponding to thetracked position and velocity of the object, analyzing the defined areaof interest to extract features indicative of the object, andclassifying the extracted features to determine the impact of the objectto the clear path; utilizing the determined impact of the object todescribe an enhanced clear path of travel; when the velocity of theobject indicates the object is in motion, identifying a clear path imagearea, comprising: determining a previously traversed path travelled bythe object based on the tracked position and velocity of the object, andidentifying the clear path image area based on the previously traversedpath, the clear path image area being devoid of other objects; andutilizing the enhanced clear path of travel and the identified clearpath image area to navigate the host vehicle.
 2. The method of claim 1,wherein monitoring two sets of sensor data describing the objectcomprises monitoring two sets of sensor data wherein each set describesa range, a time-rate of change of range, and an azimuth angle of anothervehicle.
 3. The method of claim 2, wherein utilizing the determinedimpact of the object to describe the enhanced clear path of travelcomprises removing from the clear path of travel a position of the othervehicle.
 4. The method of claim 2, wherein utilizing the determinedimpact of the object to describe the enhanced clear path of travelcomprises removing from the clear path of travel a projected futureposition of the other vehicle.
 5. The method of claim 2, whereinutilizing the determined impact of the object to describe the enhancedclear path of travel comprises affecting a clear path confidencelikelihood for different regions of the image based upon the two sets ofsensor data wherein each set describes the range, the time-rate ofchange of range, and the azimuth angle of another vehicle.
 6. The methodof claim 1, further comprising: monitoring sensor data describing aconstruction area including structures within the construction area. 7.The method of claim 6, wherein utilizing the determined impact of theobject to describe the enhanced clear path of travel comprises removingfrom the clear path of travel the structures within the constructionarea.
 8. The method of claim 6, further comprising: defining aconstruction area boundary based upon the sensor data describing theconstruction area.
 9. The method of claim 6, wherein monitoring sensordata describing the construction area comprises utilizing patternrecognition to determine lane instructions; and determining a lanegeometry restriction based upon the lane instructions.
 10. The method ofclaim 6, wherein utilizing the determined impact of the object todescribe the enhanced clear path of travel comprises affecting a clearpath confidence likelihood for different regions of the image based uponthe sensor data describing the construction area.
 11. The method ofclaim 6, wherein analyzing the tracked position and velocity of theobject comprises analyzing the tracked position and velocity of anothervehicle; and wherein analyzing the tracked position and velocity of theobject to determine the impact of the object to the clear path comprisesutilizing the tracked position and velocity of the other vehicle toaugment a clear path defined within the construction area.
 12. Themethod of claim 1, wherein monitoring the two sets of sensor datadescribing the object comprises monitoring data from a camera system.13. The method of claim 1, wherein monitoring the two sets of sensordata describing the object comprises monitoring data from a systemselected from the group consisting of a radar system, a LIDAR system, avehicle-to-vehicle communications system, a vehicle-to-infrastructurecommunications system, and an ultrasonic detection system.
 14. Themethod of claim 1, further comprising monitoring sensor data describinga lane geometry; and wherein analyzing the tracked position and velocityof the object to determine the impact of the object to the clear path isfurther based upon the lane geometry.
 15. The method of claim 1, furthercomprising utilizing the enhanced clear path of travel to augmentsubsequent monitoring of sensor data describing the object.
 16. Themethod of claim 1, further comprising utilizing the clear path detectionanalysis to describe an enhanced location of the object.
 17. The methodof claim 16, wherein utilizing the clear path detection analysis todescribe the enhanced location of the object comprises describing anenhanced location of another vehicle proximate to the host vehicle. 18.The method of claim 16, wherein utilizing the clear path detectionanalysis to describe the enhanced location of the object comprisesdescribing an enhanced location of a construction area object. 19.Method for detecting a clear path of travel for a host vehicle includingfusion of clear path detection by image analysis and detection of anobject within an operating environment of the host vehicle, the methodcomprising: monitoring an image from a camera device; analyzing theimage through segmentation-based clear path detection analysis todetermine a clear path of travel within the image; monitoring two setsof sensor data describing the object wherein each set of sensor datacomprises a range, a time-rate of change of range, and an azimuth angleof the object; fusing, over a series of discrete time events, the twosets of sensor data to determine a tracked position and velocity of theobject; analyzing the tracked position and velocity of the object todetermine an impact of the object to the clear path, comprising:extracting a defined area of interest in the image corresponding to thetracked position and velocity of the object, analyzing the defined areaof interest to extract features indicative of the object, andclassifying the extracted features to determine the impact of the objectto the clear path; utilizing the determined impact of the object todescribe an enhanced clear path of travel; when the velocity of theobject indicates the object is in motion, identifying a clear path imagearea, comprising: determining a previously traversed path travelled bythe object based on the tracked position and velocity of the object, andidentifying the clear path image area based on the previously traversedpath; and utilizing the enhanced clear path of travel and the identifiedclear path image area to navigate the vehicle.
 20. System for detectinga clear path of travel for a host vehicle including fusion of clear pathdetection by image analysis and detection of an object within anoperating environment of the host vehicle, the system comprising: acamera device monitoring a view in front of the vehicle; and aprocessing module comprising a general-purpose digital computerconfigured to execute the steps comprising: monitoring an image from thecamera device, analyzing the image through clear path detection analysisto determine a clear path of travel within the image, monitoring twosets of sensor data describing the object, each set of sensor datacomprising a range, a time-rate of change of range, and an azimuth angleof the object, analyzing the tracked position and velocity of the objectto determine an impact of the object to the clear path, comprisingextracting a defined area of interest in the image corresponding to thetracked position and velocity of the object, analyzing the defined areaof interest to extract features indicative of the object, classifyingthe extracted features to determine the impact of the object to theclear path, utilizing the determined impact of the object to describe anenhanced clear path of travel, when the velocity of the object indicatesthe object is in motion, identifying a clear path image area,comprising: determining a previously traversed path travelled by theobject based on the tracked position and velocity of the object, andidentifying the clear path image area based on the previously traversedpath; utilizing the enhanced clear path of travel and the identifiedclear path image area to navigate the vehicle.
 21. The system of claim20, further comprising a radar imaging system; and wherein the two setsof sensor data describing the object comprises two sets of data from theradar imaging system describing the object.